-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Use Proto Types in storage instead of bytes #3112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: chore/integrate-block-aggregator
Are you sure you want to change the base?
Use Proto Types in storage instead of bytes #3112
Conversation
message BlockResponse { | ||
oneof payload { | ||
Block literal = 1; | ||
string remote = 2; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think remote
should be an object so it's more flexible going forward. My initial thought it that it would look like this:
message RemoteBlockResponse {
string path = 1;
oneof config {
RemoteBlockConfigS3 s3 = 2;
RemoteBlockConfigHTTP http = 3;
}
}
message RemoteBlockConfigS3 {
string region = 1;
string bucket = 2;
bool requester_pays = 3;
optional endpoint = 4;
}
message RemoteBlockConfigHTTP {
string endpoint = 1;
optional map<string, string> headers = 2;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool. Makes sense to me.
tokio::sync::mpsc::channel::<Result<BlockResponse, Status>>(16); | ||
let (tx, rx) = tokio::sync::mpsc::channel::< | ||
Result<ProtoBlockResponse, Status>, | ||
>(16); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
16? is this arbitrary? if the client thats receiving the blocks is really slow, wouldn't that result in send failures from the spawned task?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add better documentation.
if the client thats receiving the blocks is really slow, wouldn't that result in send failures from the spawned task?
I think the task will just block until the stream is cleared. From the okio::sync::mpsc
docs:
/// # Examples
///
/// In the following example, each call to `send` will block until the
/// previously sent value was received.
///
/// ```rust
/// use tokio::sync::mpsc;
///
/// #[tokio::main]
/// async fn main() {
/// let (tx, mut rx) = mpsc::channel(1);
///
/// tokio::spawn(async move {
/// for i in 0..10 {
/// if let Err(_) = tx.send(i).await {
/// println!("receiver dropped");
/// return;
/// }
/// }
/// });
///
/// while let Some(i) = rx.recv().await {
/// println!("got = {}", i);
/// }
/// }
/// ```
```
I'm not exactly sure what the Tonic stream is doing under the hood. It may be clearing the buffer into its own buffer for us 🤷
ScriptTx script = 1; | ||
// CreateTx create = 2; | ||
// MintTx mint = 3; | ||
// UpgradeTx upgrade = 4; | ||
// UploadTx upload = 5; | ||
// BlobTx blob = 6; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of these being ...Tx
can we spell out Transaction
? It keeps it cleaner & had no impact to the traffic on the wire.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice to verify the deserilziation of the blocks
message Transaction { | ||
oneof variant { | ||
ScriptTransaction script = 1; | ||
// CreateTx create = 2; | ||
// MintTx mint = 3; | ||
// UpgradeTx upgrade = 4; | ||
// UploadTx upload = 5; | ||
// BlobTx blob = 6; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add a todo please? And create an issues to implement it. Or do you want to do that before we merge to the master?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Working on it here:
#3116
I can add an issue to be thorough.
type OwnedKey = BlockHeight; | ||
type Value = Self::OwnedValue; | ||
type OwnedValue = Block; | ||
type OwnedValue = ProtoBlock; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For ProtoBlock
we want to use protobuf-based encoder and decoder, not Postcard
// TODO: Should this be owned to begin with? | ||
let (header, txs) = block.clone().into_inner(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should work with reference, cloning block looks like an overkill.
Linked Issues/PRs
followup to feedback from #3100
Description
Checklist
Before requesting review
After merging, notify other teams
[Add or remove entries as needed]